DOCUMENT RESUME 



ED 426 634 



HE 031 715 



AUTHOR 

TITLE 



PUB DATE 
NOTE 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Johnson, Helene 

How To Promote Leadership and Participation in the Process 
of Periodic Programme Evaluation: Lessons Learned from 
Evaluation Practice in Higher Education in Quebec. 
1997-11-07 

7p . ; Paper presented at the Annual Meeting of the American 
Evaluation Association (San Diego, CA, November 1997) . 
Opinion Papers (120) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

Cooperation; Evaluation Methods; Foreign Countries; Higher 
Education; * Leadership; * Leadership Qualities; 

* Participation; * Program Evaluation; Role; Success; 
♦Universities 

Laval University (Canada) ; Quebec 



ABSTRACT 



This paper discusses the style of leadership required in an 
evaluation process as well as the conditions under which the participation of 
various groups of evaluation users may be ensured, using examples from higher 
education in Quebec (Canada) . It reviews the organization and evaluation of 
higher education institutions in the province, providing specific information 
on program evaluation at Laval University. The paper then reviews conditions 
favoring evaluation success, including institutional commitment, clear 
definitions of roles and responsibilities, clarification of what is at stake 
in regard to the evaluation process, the cooperation of those involved in the 
evaluation, promptness in the completion of the evaluation, and the 
importance of professional help and support services. It is concluded that if 
participation in the evaluation process is to be effective, shared leadership 
must be carried out during all stages of the evaluation process. It is argued 
that this type of leadership emerges in response to the dictates of the 
situation, to the competencies required and to the roles played. (MDM) 
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The key to success in any evaluation process lies in securing the co-operation of the those directly 
affected by the programme. Promoters of a style of evaluation which makes use of information 
generated by the process itself insist on placing those parties with a stake in the programme at the 
centre of the process, since ultimately it is they who will have to put the results to use (Greene, 1988; 
Patton, 1997, Fetterman, 1997). Having become aware of the importance of involving parties 
impacted by an evaluation, institutions have opted for participatory models. On the other hand, years 
of experience in the university setting have taught us that the willingness to participate is not always a 
given and that participation requires support if it is to be effective. 

Using examples drawn from the field of higher education in Quebec, this paper discusses the style of 
leadership required in an evaluation process as well as the conditions under which the participation of 
the various groups of evaluation users may be ensured. In the university setting, these users are 
administrators, professors, students and graduates, prospective employers, and government which 
assume the responsibility for financing higher education. 



Evaluation Leadership at the Provincial Level 

Quebec organizes its higher education somewhat differently than do most other jurisdictions in 
Canada and the United States. Its structure is unique in that, between high school and university, it 
offers a level of collegiate education known as the CEGEP (General and Vocational Colleges). These 
junior colleges typically offers two main streams: three year technical-vocational programmes and 
two-year pre-university programmes. 

At the organizational level, the Ministry of Education of Quebec intervenes by means of the 
budgetary regulations that govern university financing; operational grants represented 87 per cent of 
university funding in 1993 (Hamel, 1994). Since the provincial government funds universities with a 
view to maintaining accessibility in all regions and for all socio-economic groups, accountability is 
essential. 

Programme evaluation provides a rare opportunity for appraising the quality and relevance of the 
various dimensions of a training programme using the needs of individuals, of society and of the 
science in question as the basic criteria of assessment. The implementation of a programme 
evaluation policy is one way of ensuring that programmes are actually getting evaluated (CREPUQ, 
1994). The application of such a policy depends to a large extent on the leadership and support 
which the system is able to provide and on the level of participation of those affected by the 
programmes. 

While evaluation has become part of the general institutional culture of Quebec’s colleges and 
universities, it takes place within different frameworks at these two levels. Programmes at the 
collegiate level are evaluated under the leadership and supervision of the College Education 
Evaluation Commission (CEEC). The Commission is an independent government organization whose 
evaluation mandate covers most aspects of college education, with special emphasis on student 
achievement and programmes of study (Government of Quebec, 1994 and 1997). 

In contrast to the junior colleges, universities enjoy a great deal of autonomy. Carrying full 
responsibility for their programmes, each university develops its own programmes of teaching and 
research. There is, however, a special organization which deals with the creation of new programmes 
and the periodic evaluation of existing programmes at the university level-the Conference of Rectors 
and Principals of Quebec or CREPUQ. To translate this into the American scene, we might call this 
group the Association of University Presidents. This organization, funded by its member institutions, 
brings Quebec universities together on a voluntary basis. Its role involves co-ordination and 
rationalization of the programmes offered by all the universities of the Province of Quebec. The 
scope of my remarks here will be limited to the question of programme evaluation at the university 
level. 
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In 1994, CREPUQ adopted an Appraisal Policy to ensure that universities evaluate their programmes, 
using procedures which satisfy a set of generally recognized criteria and standards that correspond to 
the high expectations which society expects from universities. To further enhance the credibility of 
this initiative, the policy was to be accompanied by an external audit procedure entrusted to the 
Programme Appraisal Audit Commission or the PAAC. 

In 1997, the Association of University Presidents (CREPUQ) announced the creation of CUP (la 
Commission des universites sur les programmes or The University Programmes Board). This board 
was set up to study programme relevance and overlap among its member institutions. The university 
community agreed to undertake this task during a series of public meetings known as the Estates 
General on Education. The Minister of Education agreed in turn to take this initiative into 
consideration in any future plans related to university education. CUP’s several objectives 
complement the universities’ own efforts to rationalize their activities. For its part, the CREPUQ is 
exercising a leadership role at the provincial level as it supports a process of negotiation and trade- 
offs among its participating institutions. 



Periodic Programme Evaluation at Laval University 

Some 33 000 students are registered in one or another of the 300 programmes offered at Laval 
University. Our institution began to evaluate its programmes of study in the early 1980s. Although 
not a systematic effort, each year this operation did evaluate 5 to 10 programmes on a voluntary 
basis. In 1992, the University decided to systematize its programme evaluation, with the effect that the 
overall process was speeded up until some 30 to 40 programmes were being evaluated annually under 
the joint responsibility of the central administration of the university and the various faculties. Since 
1997, in a move intended to decentralize power and responsibility within the university, programme 
evaluation was integrated into evaluation at the faculty level (Laval University, 1997). 

The PAAC visited Laval University in 1993. As a part of the follow up to its recommendations, the 
University adopted a new periodic evaluation procedure for use in all its programmes beginning in 
1995. This new procedure gave an important role to external experts from other universities who 
were asked to determine, based on their own visit to the faculty and on the faculty’s self-evaluation 
report, whether or not the programmes being offered were relevant and of a sufficiently high quality 
to make them competitive with similar programmes offered by other universities. The changes 
implemented in the wake of the PAAC’s visit are an indicator of the impact of the leadership assumed 
by this organization in the Quebec higher education system. 



Conditions Favouring Evaluation Success 

Two years after the implementation of the new evaluation procedure, we believe that the time is right 
for a preliminary assessment which will allow us, should the need be felt, to correct any shortcomings 
in our way of doing things. Hoping to identify the strengths and weaknesses of the evaluation process 
as well as the conditions necessary for achieving effective participation, we conducted a series of 
interviews with key players. 



The Institutional Commitment 

When an institution officially undertakes a periodic evaluation of all its programmes, any and all 
mechanisms essential for it to attain its objectives must be put in place. It is not enough simply to 
build sufficient momentum to set the wheels of a massive evaluation project in motion. A genuine 
institutional commitment must be present in order to ensure that the operation is carried out, that the 
evaluations performed are of a sufficiently high quality and that recommendations emanating from 
the operation are implemented in a publicly accountable manner. 
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Laval University made a firm commitment to the process when it added an evaluation coordinator to 
its top management. This individual assumed a position of leadership, formulating directives and 
providing the means required to carry out the periodic evaluations in the time frames allotted. Deans, 
in the choosing when and what specific aspects to evaluate, exercise leadership with respect to the 
periodic evaluation of programmes within their own faculties. The administration of each faculty is 
thus responsible for establishing evaluation mandates, and at the end of the operation, for formally 
accepting to follow up on the evaluation's recommendations by proposing a plan of action to the 
University Council. 



The Clear Definition of Roles and Responsibilities 

The change in evaluation procedures gave rise to numerous reactions on the part of the groups of 
people involved. We noticed, for instance, that the participation of outside experts was generally 
looked upon with favour by those responsible for evaluation, in part because of the visibility it gave 
their programme. Resistance was experienced, however, in the selection process for members who 
were to serve on the final evaluation committee. Composed of professors who have no evident 
relationship to the programme being evaluated, since they come from outside the faculty, the role of 
this committee is to offer a synthesis of the report of the external examiners and the faculty’s own 
report. It is they who formulate any recommendations emanating from the process. In our opinion, it 
was because the role played by this committee was not well understood that it met with such 
resistance. 

One possible explanation for this mistrust was the doubt regarding the ability of non-specialists to 
pronounce an enlightened judgement on matters peculiar to the programme under evaluation, or the 
fear that they might give a biased judgment thus favouring a rival programme in the eyes of a 
potential clientele. In order to avoid such conflicts, it is necessary to clearly define the roles of the 
various participants and to see to it that the rules are applied so as to guarantee impartiality in the 
selection process for those called upon to appraise the programme. 



The Clarification of What is Really at Stake 

Another factor which likely adversely affects the involvement of the main actors has to do with a 
resistance to change. Sometimes those involved with evaluations sense that certain unavowed elements 
are coming into play in the process. While in our context, the stakes are essentially identical from 
one evaluation to the next, we were able to identify different reactions on the part of individuals who 
were concerned by a loss of autonomy with respect to programme orientation. This underlines how 
important it is that administrators make known, from the outset, limitations of the political or financial 
variety (for example: the reduction of the number of programmes offered in a particular field of 
study). Once the constraints are known the room to manoeuvre will be reduced, but any effort 
expended will be done within more realistic limits. Nothing is more disheartening for participants in 
an evaluation than to see a recommendation introduced which is not based on information generated 
by that process. Since recommendation must be made in any evaluation, it is important to clarify the 
intended uses of a particular evaluation at the very beginning of the operation. 



The Co-operation of Those Involved 

Experience has taught us the advantages of letting people know how important their participation is 
to the success of the evaluation process. Given the large number of people touched by a university 
programme, it would, however, be difficult, if not wholly unrealistic to think that a formula might be 
worked out whereby everyone gets involved at every stage in the process. It is useful therefore, to 
distinguish two different levels of participation. On the first level, participation would entail the active 
involvement of a relatively small group of participants who would form a team to oversee the 
evaluation. At a second level, involvement would take the form of a series of consultations which 
would allow the strengths and weaknesses of the programme under evaluation to be identified. 
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Unfortunately, we have seen people who show up at the very last stage of the process... This sort of 
behaviour risks slowing down the evaluation process and putting obstacles in the way of 
implementing recommendations since the report has not been appropriated by all the concerned 
players. 

Touching on another aspect of the overall process, we noticed that the leadership role which students 
are able to play is often missed. Their contribution is underestimated when it comes to active 
participation in certain strategic steps in the evaluation process, for example when dealing with the 
fine tuning and administration of the tools used for consultation as well as with the contextualization 
of the data gathered. 

Here then, are a few initiatives which we have found useful in trying to get users involved: an official 
launch of the operation, an information letter sent to faculty administrators, announcements at 
faculty-wide and departmental gatherings, invitations to student associations and finally the 
dissemination of evaluation results to those who participated in the process and in the media. The 
invitation to participate must be extended in a climate of confidence, of cordiality and of democratic 
process. 



Keeping the Pace Up 

Under old procedures, it was not unheard of for a process of periodic evaluation to stretch out over a 
three-year period. Prolonging an operation can produce both positive and negative effects. On the 
plus side, we have the time required to carefully lay the ground work and to craft the appropriate 
tools for consultation, as well as the time needed to properly gather and analyze the data. Time is also 
on the side of those seeking to open people up to new ways of looking at the programme and thus 
for preparing the ground for change. On the negative side, the stamina of those responsible for the 
operation may be found wanting and a loss of interest is a real possibility when this happens. 
Stretching the process out inevitably leads to a change in membership of the piloting team, to down 
time and to unforeseen setbacks. Limiting the time allotted for a programme evaluation within a year 
has the advantage of keeping the pace of work brisk and the morale and motivation of participants 
high. 



The Importance of Professional Help and Support Services 

The complexity of the process and the limitations linked to the length of an operation underlines the 
need for proper support, in other words, support assured by co-operation among people coming 
from different administrative units. Here are a few examples that will illustrate the variety of efforts 
made to support the evaluation process. 

Attached to the central academic administration of the University, an internal consultant specialized 
in measurement and evaluation provides help with the planning of the various stages of the evaluation 
process. This evaluation facilitator ensures the quality and confidentiality of the data gathered during 
the numerous consultations and offers follow-up services to make sure recommendations are actually 
implemented. 

The Office of Institutional Research compiles a statistical file on the clientele of the programme 
under evaluation and provides the results of the official inquiries which it regularly conducts with the 
graduates of the various programmes. Such information, made available at the outset of a periodic 
evaluation, allows one to choose relevant questions for consideration and allows these questions to be 
written into the mandate of the evaluation. 

The Student Placement Office, which deals with between five and six thousand job offers per year, 
analyzes and reports on the needs and emerging trends within the job market in those sectors of the 
economy targeted by the programme under evaluation. 
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Conclusion 

Our attempts to achieve a meaningful participation of the actors involved in the evaluation process 
have led us to enumerate some basic requirements. The participation of those involved in the process 
is influenced by a large number of factors which contribute to the degree of success achieved by a 
programme evaluation process. 

If participation is to be effective, the image of a single great leader followed by the mass of those 
involved must be abandoned. Shared leadership appears to be one condition of success, that is, a 
leadership shared among several people and spread over time, following the various stages of the 
evaluation process. Oriented towards a specific task, this type of leadership emerges in response to the 
dictates of the situation, to the competencies required and to the roles played. In short, it is a kind of 
relay leadership... 

In closing, I would like to offer a thought drawn from a book published in 1975 by Professor Henri 
Saint-Pierre from Laval University, under a title which translates roughly to: Participation, Towards 
Empowerment : 

«The most important thing is not to imagine a participation that is utopic 
but rather to believe in one’s own resources ; when the air is clear, 
difficulties get resolved, since they too become a target of responsible 
action.» 
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